An Experiment in Game-Based Classifier Selection

We present a two-player game against nature for the study of classifier combination. The game is an extension of the prediction with expert advice framework developed by Cesa Bianchi et al.. In the game, the player selects from a set of base classifiers and their combinations, playing a closed-world competitive prediction with expert advice game, with the aim of selecting one of the classifiers that will achieve the minimum error. To demonstrate our approach we present a simple game for a binary classification task using the MNIST data set. From our exhaustive evaluation of this scenario we develop two simple strategies for selecting the best forecaster. In the future, the game presented may be used to study various classification contexts, structural pattern recognition problems, and the use of learning algorithms to infer strategies for the game.